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[57] ABSTRACT 

A scene is imaged from a plurality of points of view to 
generate plural images each including an image value and 
distance for each picture element The images are converted 
to a principle point of view to generate a non-standard image 
that accounts for hidden objects in the scene from the 
perspective of the principle point of view by including 
multiple image values and distances for each picture ele- 
ment The non-standard image is then converted to generate 
a plurality of display images, each having a different display 
point of view. The display images are individually displayed 
at corresponding display elements arrayed in a display panel 
for viewing. A lens at each display element fans out the 
display image and thus restricts the amount of the image 
visible at any given perception point to produce true three- 
dimensional images. An application of the system is dis- 
closed for effectuating a virtual conferencing facility. 
Furthermore, disclosure is made of the generation or modi- 
fication of three-dimensional images in the non-standard 
image format 

50 Claims, 5 Drawing Sheets 
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TRUE THREE-DIMENSIONAL IMAGING differing perspective images are cut into a plurality of 

AND DISPLAY SYSTEM vertical columns and interleaved to form a single, combined 

image. The parallax barrier strip or sheet of parallel lenticu- 
lar lenses is overlaid on the combined images and functions 

BACKGROUND OF THE INVENTION 5 to restrict each eye to the viewing of a different one of the 

, _ . . - , _ » - _ interleaved images. The viewed images are men converged 

1. Technical Field of the Invention by the brain to give the perceptionbf depth. This system. 
The present invention relates to three-dimensional imag- however, produces images with a limited viewing angle in 

ing systems and, in particular, to the camera systems, image that movement of the viewer left or right of off-center 

data manipulation systems and image display systems nec- jo produces either a flat image or causes a reversal in the image 

essary for taking, generating, transmitting and reproducing from orthoscopic to pseudoscopic where the far and near are 

true three-dimensional images. replaced with each other. The viewing angle limitation of a 

2. Description of Related Art parallax barrier strip or sheet of parallel lenticular lenses 
The most commonly recognized example of three- becomes more pronounced when three or more differing 

dimensional image perception is that experienced with 13 perspective two-dimensional images are interleaved to form 

human vision. Although each eye views the same scene, the 411 autostereogram. Furthermore, true three dimensional 

horizontal displacement between the eyes allows each eye to perception of the imaged scene is not always possible, or is 

view that same scene and any objects located therein from limited in scope as the images used to construct the com- 

a slightly different perspective. For example, one eye will Wned image are usually taken from the same point of view 

see slightly more around a comer or curved surface of a 20 or verv ^ ew Cerent points of view, 

viewed object than the other eye. Breaking up the viewed The foregoing systems and methods for producing ste- 

scene into a plurality of picture elements reveals that cor- reoscopic three-dimensional photographic images have been 

responding elements viewed by each of the eyes are slightly extended to video or motion picture images. For example, 

displaced horizontally from one another due to the depth of the outputs of two spaced apart television cameras or motion 

(or distance to) the object The brain receives the displaced 25 picture cameras viewing the same scene have been directed 

images from each eye. and compares and combines (i.e.. to or projected on a display for stereoscopic viewing using 

converges) the images to perceive differences in distances binocular, alternating display, filtering or parallax apparatus, 

between the various objects in the viewed scene. This visual Although equally as successful as their photographic coun- 

effect is commonly referred to as stereo scopy. terparts in producing three-dimensional effects, these sys- 

Considerable effort and expense has been invested in the 30 terns also suffer equally from the same drawbacks as expe- 

development of methods and systems for processing taken rienced with three-dimensional photographic images, 

two-dimensional images so as to permit viewers to perceive including the limitation concerning the provision of only a 

the taken images in three dimensions. Such methods and single, or very limited number of points of view, 

systems commonly utilize eye-to-eye parallax to provide Accordingly, there is a need for an imaging and display 

depth information by taking two-dimensional images from 35 system and method for generating true three-dimensional 

two slightly different perspectives. The images are indlvidu- images, 
ally viewed by different eyes to produce stereoscopic effects 

by means of blinker-like shields or binocular apparatus worn SUMMARY OF THE INVENTION 

about the eyes. ^ A scene and the objects located therein are imaged from 

As an alternative to separate eye display of the two a plurality of points of view. Each acquired image comprises 
slightly different perspective two-dimensional images, the a plurality of picture elements, with the picture elements 
images are instead overlaid on each other as an analglyph including image information concerning the objects in the 
and displayed on a single screen. The images are separated imaged scene visible from the point of view of that image, 
from each other for human viewing and stereoscopic per- 45 The images of the scene are converted to a single, 
ception through the use of spectacles having a different non-standard image of the scene from the perspective of a 
coloring or polarizing filter for each eye. In yet another principle point of view. The non-standard image also corn- 
alternative, the two images are alternatively displayed on a p^ a plurality of picture elements, however, in this case 
single screen with the eyes of the observer alternatively cac h picture element is associated with a particular orien- 
occluded by spectacle mounted shutters actuated in phase x c f a scan ray emanating from the principle point of 
with the alternating image display to produce a stereoscopic v j cw> and i oc iudes image information for each object 
tftttt imaged in the scene located along that orientation of the scan 

Each of the foregoing three-dimensional imaging solu- ray. Accordingly, the non-standard image accounts fa the 
dons effectively produces three-dimensional images for presence of all imaged objects in the scene, including those 
viewing, but inconveniently requires the use of some sort of 55 objects which are not necessarily visible from the principle 
apparatus (like spectacles) to be worn about the eyes of the point of view for the non-standard image, 
viewer. A further drawback experienced with such solutions The generated non-standard image is processed and con- 
is that the viewer is limited to the point of view from which verted from the principle point of view to generate a 
the images are taken for experiencing the three-dimensional plurality of display images of the scene from the perspective 
effect True (i.e., multi-point of view) three-dimensional go 0 f a corresponding plurality of display points of view. Each 
viewing where the sides of objects as well as objects hidden display image comprises a plurality of picture elements, 
behind other objects are revealed by lateral viewer move- with each picture element comprising image information for 
ment with respect to the taken images is not possible using only those objects in the scene visible from the correspond- 
any of the foregoing methods. ing display point of view. The plurality of display images are 

Three-dimensional images may be produced without use 65 presented for viewing by an arrayed plurality of display 

of viewer worn apparatus by means of a parallax barrier strip elements in a display panel, each included display element 

or sheet of parallel lenticular lenses. In such a system, the corresponding to one of the plurality of display points of 
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view. Each display element includes a lens for fanning out in accordance with the present invention. The system 10 

the display image and thus limiting the number of picture includes an image acquisition device 12 for simultaneously 

elements in the display image visible from any one given taking full field of view 18 images of a scene 14 (containing 

perspective of a viewer. As the viewer moves laterally with objects 16) from multiple points of view (not shown, see 

respect to the display panel and the perspective accordingly 5 FKJ. By "objects*' 16 it is meant to include not only 

changes, different picture elements in the plurality of display distinct objects (e.g.. two trees), but also distinct features or 

images become visible through the lenses of the display of a single object (e.g., two leaves on a tree, or two 

elements thus allowing different points of view of the scene different features on a single leaf). The taken multiple point 

to be perceived by the viewer in three dimensions. vicw images are converted (composed) by the image acqui- 

10 sition device 12 to generate a single, non-standard, full field 

BRIEF DESCRIPTION OF THE DRAWINGS of view 18 image of the scene 14 and all its included imaged 

A more complete understanding of the method and appa- objects 16 (as well as backgrounds in the scene 14 — not 

ratus of the present invention may be obtained by reference shown) from the perspective of a chosen principle point of 
to the following Detailed Description when taken in con- 

junction with the accompanying Drawings wherein: 15 The non-standard image is output from the image acqui- 

FIG. 1 is a block diagram of a true three-dimensional sition ****** n on communications link 22 for further 

imaging and display system in accordance with the present processing and true three-dimensional display as will be 

invention * described in more detail herein. The communications link 22 

FIG. 2 illustrates a scan ray for ^-dimensional image ? °/ n ^ e \ ekhcr a wirel ^ c ^ a ^twork (WAN, LAN, 

acquisition* 20 tele P none ) communications link, or a radio frequency 

4 . . - . . . . ^ (television, microwave) communications link as needed and 

FIG. 3 lUustrates the scanning of a hemisphere by the scan having sufficient bandwidth for me particular application of 

n-ytfITO.2; the system 10. 

FIG. 4 is a block diagram of a true three-dimensional Reference is now made to FIGS. 2 and 3. By 'lull field of 

image acquisition apparatus; ^ ^cw* 18 imaging it is meant that image information is 

FIG. 5 illustrates the ray projection process utilized to acquired by the image acquisition device 12 from each point 

convert non-principle point of view images to the principle of view by scanning a scan ray 24 over the scene 14 one 

point of view; hundred eighty degrees in both the horizontal 26 and vertical 

FIG. 6 is a block diagram of a stereo camera in the image 28 directions. Imaging in this manner results in the scanning 

acquisition apparatus; 30 of a three-dimensional space represented by a hemisphere 

FIGS. 7 A and 7B are diagrams representative of the shape 30. The hemisphere 30 is scanned from left to right along 

of the images generated by the cameras; horizontal arcs 32 in raster fashion from top to bottom. Each 

FIG. 8 is a block diagram of a true three-dimensional ima S c of 0x5 scene 14 acquired by full field of view 18 

image storage and manipulation device; imaging from any of the multiple points of view comprises 

FIG. 9 shows an orthogonal view of a true three- 35 a plurality of picture elements, with each picture element 

dimensional image display panel; associated with a particular orientation of the scan ray 

FIG. 10 is a block diagram of an individual display 24 f wi * «* clcmcnt kdudi ^ ^ inf *" 

element; mation for the scene 14 along the corresponding scan ray 24 

rtp I, . . . , . ... . _ orientation (see, FIG. 1). 

FIG. 11 is a cross-sectional view of the bubble lens for ... t .... 

each display element' 40 n wlU te notcd wim rastcr scanmn g of &e henu - 

FIG. 12 is a block diagram of an individual laser source fP^ 30 will be morepicture elements along arcs 32 

in the display dement cf FIG. 10; locatcd * c «** <* me hemisphere than there are 

CT „ - . picture elements along arcs located closer to the top and 

HG. 13 illustrates the reverse ray projection process bottom of the hemisphere. Theoretically, the top and bottom 

uulized to convert principle point of view images to other 45 MCS 32 wl]1 ^ ODl a single picture element As a matter 

pouits of view inthe display; of practice, however, the first and last few arcs 32 and side 

HG. 14 is an orthogonal view of the installation of the e4gesofthe hemisphere 30 may not be scanned at all by the 

present invention in a room for implementing a teleconfer- image acquisition device because it is unlikely that such 

encing application; information will be either useful to a viewer or used in 

FIGS. ISA and 15B are top down schematic views jq subsequent image manipulation or display, 

illustrating the assembly of a plurality of separate rooms in Referring now also to FIG. 16, each image may be 

which the present invention is installed to define a 'Virtual" represented as a frame 200 of data. When multiple images 

conference room; axe taken and sequentially output (for example, in the case 

FIG. 16 illustrates the format of the data for output 0 f video imaging), corresponding multiple frames 200 of 

images; 55 data are sequentially generated (see, FIG. 19 A). Each frame 

FIG. 17 illustrates the format of the pixel field for a 200 of data includes a frame start (FS) 202 followed by a 

conventional standard image; plurality of pixel fields (PF) 204 each containing image data 

FIG. 18 illustrates the format of the pixel field for a for a corresponding picture element in the image. For a 

non-standard image in accordance with the present inven- conventional or standard image, each pixel field 204 

tion; 60 includes data relating to the color 208 (red, green and blue 

FIGS. 19A and 19B illustrate the format for the data in a color image; or gray scale in a monochrome image) and 

transmission of video images; and intensity 210 of the picture element for the closest imaged 

FIG. 20 illustrates the format for the image value field. °bfert ti« corresponding scan ray 24 (see, FIG. 17). 

nPTAn nn np^nmnM rw pviRnnTMPvrc Because raster scanning is being performed, the first pixel 

DETAILED DESCRIPTION OF EMBODIMENTS ^ field 204 in the frame 200 contains the image data for the top 

Referring now to FIG. 1. there is shown a block diagram and left-most picture element in the image. The next pixel 

of a true three-dimensional imaging and display system 10 field 204 contains the image data for the next picture 
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element, and so on. arc 32 by arc through the hemisphere 30, Reference is now made to FIG. 4 wherein there is shown 
until the last pixel field is reached containing the image data a block diagram of the image acquisition device 12 com- 

for the bottom and right-most picture element in the image. prising a plurality (two shown) of stereo cameras 38 posi- 

A frame end (FE) 206 may, but need not necessarily, follow tioned at selected different, non-principle points of view 40 

the last pixel field 204 signifying the end of the frame 200 5 and connected to a three-dimensional imaging processor 42 

of data for that one image. via communications links 44. Each of the included stereo 

Referring now again to FIG. 1. and also to FIG. 18, the cameras 38 takes and outputs full field of view 18 images of 
non-standard, full field of view 18 image of the scene 14 the scene 14 from its particular point of view 40. The images 
from the principle point of view 20 generated and output by output by the stereo cameras 38 have the format illustrated 
the image acquisition device 12 has the same data format io in HG. 16. Accordingly, each image comprises a plurality of 
including a sequential plurality of pixel fields 204 as illus- picture elements, with each picture element being associated 
trated in FIG. 16. Accordingly, each image comprises a with a particular orientation of a non-principle scan ray 24n 
plurality of picture elements as discussed above, with eacfi emanating from the point of view 40 of the stereo camera, 
picture element being associated with a particular orienta- Because the imaging performed is conventional in nature, 
tion of a principle scan ray 2Ap emanating from the principle 15 each picture element output from a camera 38 has a pixel 
point of view 20. In the case of this non-standard image, field 204 defined by only a single image and distance value 
however, instead of being defined by image information for pair 212. The included image value 214 in the pair provides 
only the nearest object as is conventional, the pixel field 204 image data (e.g., color and intensity) for the nearest object 
for each picture element is instead defined by at least one 16 located along the particular orientation of the non- 
(and often more than one, as shown) image and distance 20 principle scan ray 24n corresponding to that picture element, 
value pair 212. The image value (IV) 214 in each included The associated distance value 216 in the pair provides a 
pair 212 provides image data (e.g., color 208 and intensity distance to that nearest visible object 16 along the non- 
210) relating to an object 16 in the scene 14 located along principle scan ray 2An emanating from the location of the 
the particular orientation of the principle scan ray 24p point of view 40. No information (image or distance) is 
corresponding to that picture element Hie associated dis- 25 obtained for the sides of objects and the hidden objects 
tance value (DV) 216 in each included pair provides a (including backgrounds) which are obstructed from view by 
distance (most likely expressed in a logarithmic, the nearest visible objects imaged at the non-principle points 
exponential, or other non-linear scale) to that object 16 along of view 40. Image information for these objects is instead 
the principle scan ray 24p emanating from the location of the obtained by stereo cameras 38 positioned at other non- 
principle point of view 20. 30 principle points of view 40. 

Multiple image and distance value pairs 212 for a single The three-dimensional imaging processor 42 receives the 
picture element are generated in the output non-standard images (i.e.. image and distance value pairs 212 for the pixel 
image for a single orientation of the principle scan ray 24p fields 204 of the picture elements) output by the plural stereo 
(Lc, in a single pixel field 204) to account for instances cameras 38. and through a process referred to herein as "tray 
wherein multiple objects 16 in the scene 14 are located along 35 projection" converts the plural images of the scene 14 taken 
a particular orientation of the principle scan ray 24p (as from the non-principle points of view 40 to generate the 
generally indicated at 34 in FIG. 1) extending from the single, non-standard, full field of view 18 image of the scene 
location of the principle point of view 20. For example, from the perspective of the principle point of view 20. In the 
given the orientation of the principle scan ray 24p illustrated ray projection process, each orientation of a non-principle 
at 34, the pixel field 204 for me m'cture element correspond- 40 scan ray 24n is matched (as closely as possible) to a 
ing to that orientation includes one image and distance value corresponding orientation of the principle scan ray 24p. 
pair 212 for each of the two objects 16 located along the scan Thus, picture elements and pixel fields 204 in one image 
ray, as well as a third image and distance value pair for the from the non-principle point of view are matched to picture 
background (not shown) behind the objects. Thus, even elements (and pixel fields) in the image from the principle 
though not visible from the principle point of view 20, image 45 point of view. Next, the image and distance value pairs 212 
and distance values 214 and 216 for the sides 36 of objects associated with each of the orientations of the non-principle 
16 and the hidden objects (including backgrounds) located scan rays 24/t (pixel fields 204 for the picture elements) are 
behind visible objects are obtained by the image acquisition associated with the matching orientations of principle scan 
device 12 from the other points of view (not shown). ray 2Ap (pixel fields for the picture elements). This process 
converted by the image acquisition device to the principle 50 of matching the orientations of the scan rays 24 and asso- 
point of view, and then output in the generated non-standard dating the image and distance value pairs to principle scan 
image to provide a true three-dimensional image of the ray orientations is repeated across the full field of view 18 
scene 14. In order to differentiate one pixel field 204 from to obtain image information for all orientations of the 
a next pixel field, and thus associate image and distance principle scan ray 24/7. The generated and output non- 
value pairs 212 with the correct picture element each pixel 55 standard image resulting from this process has the format 
field is formatted to include a start field (SF) 218, and may illustrated in FIGS. 16 and 18. and thus includes image 
also include an end field (EF) 220. Because more than one information for all objects 16 imaged by the plurality of 
image and distance value pair 212 may be output on com- stereo cameras 38 (including hidden objects, backgrounds 
muni cations link 22 for each picture element (i.e.. principle and object sides not necessarily visible from the location of 
scan ray 2Xp orientation and/or pixel field 204), it will, of 60 the principle point of view 20). 

course, be understood that the generated non-standard image It will, of course, be understood that often times the 

is unconventional in nature and is therefore not readily various orientations of the principle and non-principle scan 

displayable for viewing using conventional means. As will rays 2Ap and 24/i at a given object 16 will not match exactly, 

be discussed in connection with FIGS. 9-13, viewing of the When this occurs, one available option is to select the closest 

non-standard image requires special handling of the 65 orientations of the scan rays 24 as matches. The image 

instances wherein a single picture element includes more information in the pixel fields 204 for those closest orien- 

than one image and distance value pair. tations is then associated as described above. Alternatively, 
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the image information (pixel fields 204) associated with all 
of the closely, but not necessarily exactly, matching orien- 
tations of the non-principle scan rays 2An is combined to 
generate a combination image and distance value pair 212 
that is associated with the determined matching orientation 
of the principle scan ray 24p and its pixel field 204. The 
combination of image information from several picture 
elements is performed in accordance with well known pixel 
correlation and processing techniques. In this case, as shown 
in FIG. 20, the image value 214 of the image and distance 
value pairs 212 may also include data relating to percentage 
226 that the image information from the non-principle point 
of view 40 fills the image information of the concerned pixel 
field 204 for the principle point of view 20. 

Referring now to FIG. 5, three objects 16(1)-16(3) are 
shown being imaged from two non-principle points of view 
40 by the illustrated orientations of non-principle scan rays 
24n. The image information generated for these orientations 
of the non-principle scan rays 24« comprises one image and 
distance value pair per illustrated orientation. In accordance 
with the ray projection process, a determination is first made 
of the particular orientation of the principle scan ray 24p 
emanating from the principle point of view 20 matching to 
the orientations of the non-principle scan rays 24n. Next the 
measured distances (d) to the objects 16(1}-16(3) from the 
non-principle points of view 40 along scan rays 24n are 
converted to distances (d') along the identified matching 
orientation of the principle scan ray 2Ap emanating from the 
principle point of view 20. This distance correction accounts 
for the displacement of the locations of the non-principle 
points of view 40 away from the location of the principle 
point of view 20. The original image values along with the 
converted distances for the objects 16{1}-16(3) are then 
associated with the illustrated orientation of the principle 
scan ray 24p. Thus, in the generated non-standard image 
output from the processor 42, the ray projection process 
produces a picture element for the illustrated orientation of 
the principle scan ray 24p that includes at least three image 
and distance value pairs corresponding to the three objects 
16(1)-16(3). as well as a fourth image and distance value 
pair for the background behind the three objects. To generate 
the non-standard image for the full field of view 18, the 
foregoing process is repeated across the full field of view 18 
such that at least one image and distance value pair is 
generated for every orientation of the principle scan ray 24/?. 

Referring now again to FIG. 4, although a stereo camera 45 
38 need not necessarily be positioned at the principle point 
of view 20 (resulting in virtual imaging from the perspective 
of the principle point of view 20 as discussed above), 
placement of a stereo camera 38 at that location (as generally 
illustrated by broken lines in FIG. 4) may facilitate the 
generation of the non-standard image. The stereo camera 38 
positioned at the principle point of view 20 outputs an image 
and distance value pair for the visible objects along every 
orientation of the principle scan ray 24p in the full field of 
view 18. Advantageously, a majority of the objects within 
the scene 14 are imaged by the stereo camera 38 from the 
principle point of view 20, and thus the image information 
relating to these objects need not necessarily be obtained by 
other stereo cameras 38. Only (hat limited amount of image 
information collected by the stereo cameras 38 at the non- 
principle points of view 40 and relating to any hidden 
objects, backgrounds or sides of objects in the scene 14 not 
visible from the perspective of the principle point of view 20 
need be collected and processed into principle point of view 
image information by the processor 42 in accordance with 
the ray projection process to generate the non-standard 
image. 



Referring now to FIG. 6, there Is shown a block diagram 
of the stereo camera 38 comprising a left camera 46 and a 
right camera 48 connected via communications links 50 to 
a distance processor 52. The cameras 46 and 48 each include 
a bubble (or fish-eye) lens 54 to facilitate imaging of the 
scene 14 over the full field of view 18. It will of course be 
understood that lenses with smaller fields of view may be 
used if desired for the lens 54. The cameras 46 and 48 are 
slightly displaced in a horizontal direction from one another 
by a predetermined displacement distance (dd) to induce 
parallax in the images taken by the cameras. The midpoint 
between the cameras 46 and 48 identifies the location 56 of 
the particular point of view of the stereo camera 38. 

The distance processor 52 receives a full field of view 
image taken by each of the left and right cameras 46 and 48. 
respectively, in the format illustrated in FIGS. 16 and 17. 
and processes the images to determine a distance (d) along 
the scan ray 24 emanating from the location 56 to each of the 
nearest visible imaged objects 16 over the full field of view 
18. The distance processor 52 first finds a common picture 
element (pixel field 204) between the two taken images and 
identifies the imaging rays rl and r2 emanating from the 
cameras 46 and 48 and pointing at that common picture 
element. The imaging rays rl and r2 along with the dis- 
placement distance (dd) define a triangle. The orientation 
(i.e., compound angle) of the imaging rays rl and r2 and the 
displacement distance (dd) are known quantities from which 
the distance (d) to the visible object 16 at the common 
picture element may be determined by the processor 52 
using trigonometry. This distance calculation is made by the 
processor 52 for each common picture element identified 
between the two taken images. The distance value 216 (see 
FIG. 18) comprises the distance (d) to the object which will 
most likely be converted to a logarithmic, exponential or 
other non-linear scale because a difference of one foot is 
more significant to the three dimensional effect for close 
objects than it is for distant objects. The processor 52 thus 
generates and outputs an image and distance value pair 212 
for each picture element comprising the picture element 
40 image value 214 and the calculated distance (d) value 216 in 
association with each corresponding orientation of the scan 
ray 24 (pixel field 204). 

It will, of course, be understood that often times the 
orientations of the imaging rays rl and r2 at a given object 
16 will not match exactly with each other to define the 
orientation of the scan ray 24 and produce the image 
information. When this occurs, the image Mormation asso- 
ciated with all of the closely, but not necessarily exactly, 
rnatching orientations of the imaging rays rl and r2 can be 
so combined to generate a combination image and distance 
value pair 212 that is associated with the scan ray 24. The 
combination of image information from several picture 
elements (pixel fields 204) is performed in accordance with 
well known pixel correlation and processing techniques. In 
this case, as shown in FIG. 20, the image value 214 of the 
image and distance value pairs 212 may also include data 
relating to the percentage 226 that the image information 
from the non-principle point of view 40 fills the image 
information of the concerned pixel field 204 for the principle 
point of view 20. 

Reference is now made to FIGS. 7A and 7B wherein there 
are shown diagrams representative of the general shape of 
the images generated by the cameras 46 and 48 of the scene 
14. As illustrated in FIG. 7A. if conventional imaging 
cameras are used for the cameras 46 and 48 of the stereo 
camera 38 a rectangular image 58 is generated. In FIG. 7B, 
the use of custom bubble lens imaging cameras for the 
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cameras 46 and 48 of the stereo camera 38 generates a 
circular image 60. The images 58 and 60 each comprise a 
circular portion 62 wherein imaging information concerning 
the scene 14 is contained as focused thereon by the bubble 
lens 54. The rectangular image 58 further includes a remain- 5 
ing area 64 containing no image information at all concern- 
ing the imaged scene 14. 

Cameras 46 and 48 that are custom designed for bubble 
lens application will efficiently operate to scan only in. and 
generate image data only for the circular portion 62 as 10 
shown in FIG. 7B. This data is transmitted over communi- 
cations link 50 to the processor 52. Conventional cameras 46 
and 48 adapted to use a bubble lens 54. however, will scan 
the entire image 58 as shown in FIG. 7A. and thus will 
generate image data for the remaining area 64 as well as the 1S 
circular area 62. In order to efficiently process the data 
output from a conventional camera, the distance processor 
52 is programmed to ignore image data for the remaining 
area 64 and only processes that image information associ- 
ated with the circular portion 62 of the image 58. 20 

Reference is now again made to FIGS. 4 and 6. The 
communications link 44 connecting the three-dimensional 
processor 42 to the distance processor 52 of the stereo 
camera 38. and the link 50 connecting the cameras 46 and 
48 to the distance processor, each comprise a uni-directional 25 
image channel for carrying image information. In the link 
44. such information comprises image and distance value 
pairs transmitted from the distance processor 52 to the 
three-dimensional processor 42 for each orientation of the 
scan ray 24 in the full field of view 18. In the link 50. such 30 
information comprises an image value transmitted from the 
cameras 46 and 48 to the distance processor 52 for each 
orientation of the scan ray 24 in the full field of view 18. 

The links 44 and 50 may also include a control channel 
over which control commands are transmitted between the 35 
various included components of the image acquisition 
device 12. Such control commands are used to direct the 
operation of the components of the image acquisition device 
12 in its processing of the images obtained of the scene 14 
by the stereo cameras 38. The commands are especially 40 
useful in directing the operation of the stereo cameras 38 
positioned at locations other than the location of the prin- 
ciple point of view 20 to acquire image data concerning 
hidden objects and the sides of objects not visible from the 
principle point of view. For example, the three-dimensional 45 
processor 42 may use the control channel of links 44 and 50 
to command the cameras 46 and 48 to acquire image 
information concerning hidden objects and sides by direct- 
ing camera imaging along certain orientations of the prin- 
ciple scan ray 2Ap emanating from the principle point of 50 
view 20. Alternatively, the three-dimensional processor 42 
may utilize the control channel to instruct the distance 
processor 52 to process and/or transmit Image information 
concerning only certain portions of the scene 14 (for 
example, image information along certain orientations of the 35 
principle scan ray 24p) to supplement the image information 
acquired from the principle point of view 20 and thus 
account for hidden objects and object sides. 

The images of the scene 14 may be acquired by the image 
acquisition device 12 in either color or black and white. 60 
When color imaging is used, it will be understood that the 
image value output from the cameras 46 and 48 or included 
in each image and distance value pair associated with a 
particular orientation of the scan ray 24 and output from 
either the distance processor 52 or the three-dimensional 65 
processor 42 includes a red value, a green value and a blue 
value. These color values comprise intensity levels for the 



10 

light in each of the particular primary colors necessary for 
producing a particular color for and intensity of the picture 
element relating to the imaged object. Monochromatic 
imaging, on the other hand, generates a single image value 
(gray scale) for each picture element 

It will further be understood that the image acquisition 
device 12 may be used to acquire either individual true 
three-dimensional images (Le., photographs) or sequences 
of true three-dimensional images (i.e., videos). The acqui- 
sition of video images from the multiple points of view 40 
and their conversion through the use of ray projection 
processing to the principle point of view 20 need not 
necessarily be performed in real time unless instantaneous 
subsequent three-dimensional (i.e., live) recording and dis- 
play of the video images is desired. As discussed above, the 
placement of a stereo camera 38 at the location of the 
principle point of view 20 may facilitate the real time 
processing required for true three-dimensional video imag- 
ing of live events. 

The data format for video images output by the image 
acquisition device 12 (FIG. 1) comprises a plurality of the 
frames 200 (FIG. 16) output in a sequence. Typically, as 
shown in FIG. 16 each frame 200 comprises a pixel field 204 
for each picture element in the images. Thus, the video 
images have a format as shown in FIG. 19A comprising a 
sequence of full frames 200. However, this is a substantial 
amount of data requiring a very high bandwidth transmis- 
sion. To reduce the bandwidth requirements, as illustrated in 
FIG. 19B, in an alternative format the data transmission 
includes periodic transmissions of frames 200 comprising 
pixel fields 204 for each picture element in an image. The 
data transmitted between these "full" frames 200 comprises 
partial frames 222 including pixel fields 204 for only those 
picture elements in the image that have changed. By 
"changed" it is meant that the image information for that 
picture element has changed from the prior image. In order 
to identify the particular picture element to which a pixel 
field 204 in a partial frame 222 belongs, each pixel field 
includes a location 224 (in either coordinates or relative 
measure) of the picture element in the image that has 
changed. Also, to further reduce bandwidth requirements, 
data compression may be performed in accordance with well 
known data compression techniques. 

Reference is now again made to FIG. 1 wherein the true 
three-dimensional imaging and display system 10 of the 
present invention further includes an image storage and 
manipulation device 66 connected to the communications 
link 22 via a bi-directional communications link 68. The 
image storage and manipulation device 66 receives and 
manages the storage of the non-standard images output on 
communications link 22 from the image acquisition device 
12. The image storage and manipulation device 66 further 
functions to facilitate user manipulation of the stored non- 
standard images. Such manipulation includes the modifica- 
tion of images in. the addition of images to, or the deletion 
of images from the device 12 output non-standard images. 

For modified and added images, the image storage and 
manipulation device 66 functions in response to user input 
to create the desired images in true three-dimensional format 
(like that generated by the image acquisition device 12 and 
illustrated in FIGS. 16. 18 and 19A-19B). Thus, each 
created image will comprise a plurality of picture elements, 
with each picture element associated with a particular ori- 
entation of the principle scan ray 24p. To account for hidden 
objects and the sides of objects, and thus provide for a true 
three-dimensional image, each picture element may, and is 
likely to, include plural image and distance value pairs as 
discussed above. 



07/19/2004, EAST version: 1.4.1 



5,675,377 

11 12 

A block diagram of the image storage and manipulation the image acquisition device 12. In such an application, 
device 66 is shown in FIG. 8. A saving and playback global location coordinates (x, y. z. pitch, roll, and yaw) or 
processor 70 connected to the bi-directional communica- similar coordinates would be needed for each image cap- 
tions link 68 functions to control the operation of the image tured by the image acquisition device 12, From this database 
storage and manipulation device 66 for saving non-standard 5 of image information, true three-dimensional images corre- 
images to. and retrieving non-standard images from a mass sponding to the particular simulation (for example, terrain in 
data storage device 72. The saved images include images a fli ^ t or ^ simulator) are generated and manipulated 
received from the image acquisition device =12 as well as m res t0 user ^ 

images created by, modified by, or added to by a simulation ^ * 

and generation processor 74. A Reference is now again made to FIG. 1 wherein the true 

« . , . . ^ - A . 10 three-dimensional imaging and display system 10 of the 

The simulation and generation processor 74 comprises a ^ . , . \_ . , 

user workstation or other processing device responsive to ulclu ^ s / f Aree-dtoeKional 

user input and commands for creating images tatfae non- T\ ge ^ 76 connected to the image acqiusition 

standard, true three dimensional format (Ha 18) previously 12 " Dd * e ™*P>*°*™ **** ** 

A^^t , f „ f „ . • ^AifiL+;~ n a! via a communications link 78. The communications link 78 

described. In addition to use in image modification, the , . . ... ... , XT _ A . r 

. , . . . . . . 15 comprises either a wireline link, a network (WAN, LAN, 

created images may comprise newly generated complete ^ . \ . . ^7; y. , 

c . j « z~ r: . J D . . _ . telephone) communications link, or a radio frequency 

images for use independent of any images received from the x ,• . * 7 

. . . . . *~ £J x it* « (television, microwave) communications link as needed for 

miage acquisition device IX Thus, the simulation and ^"«wu iiuuuwa^, tuM^uo ucww 

* . n nA . , . . . . - the particular application of the system 10. 

generation processor 74 is useful in one application for x rr ' 

creating individual non-standard images for three- M As snowi1 m 9 » me display device 76 comprises a 

dimensional display (perhaps as a sequence of images in a display panel 80 of a preselected size and shape (rectangular 

video). Alternatively, the images created by the simulation shown) including a plurality of display elements 82 arrayed 

and generation processor 74 comprise images added to in a column-by-row matrix. A display processor 84 is 

existing images stored in the data storage device 72 and/or connected to control actuation of the individual display 

received from the image acquisition device 12. Furthermore. „ elements 82 such mat a person 86 looking at the display 

the simulation and generation processor 74 operates to panel 80 will perceive a true three-dimensional image of the 

modify existing non-standard images to perform image scene nmch like mat experienced by lookmg out at the scene 

correction or deletion through a window. Although illustrated as a single unit, it 

The non-standard image creation function performed by ^ J* ^derstood that the display processor 84 connected 
the simulation and generation processor 74 of the image 30 * C0mmunicatl0ns ™k 78 may instead comprise a 
storage and manipulation device 66 facilitates use of the true ?? Processors arranged in a predetermined hierar- 
three-dimensional imaging and display system 10 as a chy within a distributed processing architecture, 
previewing device to assist in the visualization of articles F^ermore, *" ! display panel 80 need not be flat as 
prior to construction. For example, an automobile manufac- ^ated. but rather may take on a curved surface and/or be 
turer could utilize the functionality of the simulation and 35 of any selected shape as needed by the particular application, 
generation processor 74 to modify a non-standard, true three Referring now to FIG. 10, there is shown a block diagram 
dimensional image of a current model of an automobile to of an individual display element 82 in the display panel 80 
preview the appearance of the next year's model. Similar for displaying a true three-dimensional image. The display 
image modifications could be used in previewing operations element 82 includes a bubble (fish-eye) lens 88 and an 
concerning building remodeling and reconstruction, plastic 40 associated very small image display comprising a circular 
surgery, h airs ty ling and cutting, and clothing and apparel shaped microscopic laser array 90 manufactured of a plu- 
designing. Such previewing operations could further com- rality of individual laser sources 92 arrayed in a column- 
prise the creation of a true three-dimensional image of an by-row matrix. It will, of course, be understood that lenses 
article from scratch, rather than the modification of an with smaller fields of view may be used for the lens 88 if 
existing image, when a completely new design is being 45 desired The laser sources 92 each output a display ray 94. 
considered. The plurality of display rays 94 (comprising together the 

The simulation and generation processor 74 is further displayed image) output from the array of laser sources 92 

useful in connection with modifying true tridimensional m fanned out by the bubble lens 88 to cover a full (one 

video images to correct (prior to public viewing) any errors hundred eighty by one hundred eighty degree) field of view 

in the imaging process performed by the image acquisition 30 w - M illustrated in FIG. 11, spreading of the fanned 

device 12. Furthermore, the processor 74 facilitates the individual coUimated laser display rays 94 is accomplished 

creation of special effects for addition into a previously bv P^^Z *e rays through a plurality of miniature lenses 

taken true three-dimensional video image. The processor 74 102 on mc surface of the bubble lens 88. 

also enables true three-dimensional simulation of or t rainin g The number of individual laser sources 92 in the laser 

in dangerous, expensive or unique and exciting activities 55 array 90 for each bubble lens 88 is preferably equal to the 

without actual participation or cost by generating video number of different orientations of the principle scan ray 24p 

images of the activity for user viewing. Such simulation or (see. FIGS. 1 and 4-6) obtained by the image acquisition 

training activities are particularly useful in flight instruction device 12. Thus, there exists one laser source 92 for each 

because current flight simulators do cot provide the true picture element in an image displayed by the display ele- 

three-dimensional images necessary for an accurate simu- 60 ment 82. The use of fewer laser sources 92 in each array 90 

lation. User interaction with the simulation or training . of the display elements 82 is permitted, but will adversely 

images to control operation of the simulation is provided affect the overall resolution of the image by the display panel 

through an appropriate user input (e.g.. joystick, steering 80. 

wheel etc.) connected to the processor 74 workstation. The intensity of the display ray 94 output by each laser 

In a simulator application, the simulation and generation 65 source 92 is controlled by the display processor 84. 

processor 74 is also useful in building a gaming area Preferably, the intensity of the laser light emitted from each 

database from true three-dimensional images captured by laser source 92 is controlled by directly controlling the 
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operation of the laser source. Alternatively, as shown in FIG. 
12, in the event each individual laser source 92 is not directly 
controllable as to its intensity, each laser source 92 is 
constructed of a sub-array 90 of microscopic laser sources 
92' dial are individually and selectively actuated ("on" or 
"off) by the processor 84. The plurality of laser beams 94' 
output by the laser sources 92' are summed as to intensity 
and collimated by a lens system 88' to generate and output 
the display ray 94. With selective actuation of the laser 
sources 92', the intensity of the output display ray 94 is 
controlled by the processor 84. For example, the individual 
sources 92' in an sixteen-by-sixteen sub-array 90' could be 
selectively actuated to provide 256 different intensity levels 
for the display ray 94 emitted from each laser source 92. 

To produce a monochromatic display, the laser sources 92 
included in each laser array 90 output the same selected 
wavelength(s) of light To produce a color display, on the 
other hand, one option would be to have three bubble lenses 
88 with associated laser arrays 90 for each display element 
82. The three laser arrays 90 would separately emit laser 
light in the three primary colors (red, green and blue) for 
each display ray 94 with the display processor 84 controlling 
the intensity of the colors in the display ray 94 emitted by the 
laser sources 92 to produce a combined light in a selected 
portion of the visible spectrum. Alternatively, a single 
bubble lens 88 and associated laser array 90 would be used 
for each display element as shown in the display of FIG. 10. 
However, each laser source 92 in such a single laser array 90 
would comprise three sub-arrays 90*. one for each of the 
three primary colors, like that shown in FIG. 12. By con- 
trolling the intensity of the colored light emitted by the 
sub-sources in each laser source 94. the display processor 84 
would control the color of the emitted display ray 94. In the 
event red. green and blue laser light cannot be produced, 
filters in those colors could be used. Also, in an extreme 
case, it may become necessary, because of the available laser 
light colors, to choose a different set of primary colors than 
red. green and blue for the color display. In such a case, the 
display processor 84 would further perform primary color 
conversion, or the images would be acquired in the corre- 
sponding display colors. 

Reference is now made to both FIGS. 9 and 10 for an 
explanation of the operation of the display processor 84 in 
processing the non-standard image (FIGS. 16. 18 and 19B) 
received on communications link 78 to produce a true three 
dimensional image for viewing on display panel 80. It will 
be remembered that the non-standard image comprising 
multiple image and distance value pairs 212 for each picture 
element is generated by converting images taken from 
multiple points of view to a principle point of view using a 
ray projection process. The display processor 84 essentially 
performs a reverse ray projection process on the non- 
standard image to produce an image of the scene 14 from the 
perspective of a slightly different point of view for each one 
of the plurality of display points of view provided by the 
display panel 80. In fact it is preferred, for reasons of 
providing the best possible resolution in the displayed true 
three-dimensional image, that each display element 82 on 
the display panel 80 be considered as providing a separate 
display point of view for which the processor 84 determines 
and generates a unique image from the received non- 
standard image of FIGS. 16. 18 and 19B using reverse ray 
projection. Each such image that is generated by the display 
processor 84 is displayed (one picture elemcnt-^>ixcl field 
204 — in the image per laser source 92 in the display element 
82) by the display element 82 corresponding to the processor 
determined display point of view for that image. 
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To facilitate the reverse ray projectioD process, a particu- 
lar one of the display elements 82 in the display panel 80 is 
selected as the display element located at the principle 
display point of view 98 (preferably located at or near the 

5 center of the display panel). This principle display point of 
view 98 for image viewing corresponds to the principle 
point of view 20 of the image acquisition device 12 (FIGS. 
1 and 4), Thus, the non-standard image received by the 
display processor 84 from the perspective of the principle 

!0 point of view 20 is ready for display by the display element 
82 corresponding to the principle display point of view 98. 
However, only the image value for the nearest object along 
the scan ray 24 for each picture element of the non-standard 
image is displayed by the laser source 92 corresponding to 

!5 that picture element for the principle display point of view 
98. The processor 84 ignores any other image and distance 
value pairs for the picture elements because that image 
information concerns hidden objects, background behind 
objects and sides of objects that are not visible in the scene 

20 from the principle point of view 20. and thus not visible from 
the corresponding principle display point of view 98. All 
image and distance value pairs, including the foregoing 
ignored image and distance value pairs, however, are pro- 
cessed in accordance with the reverse ray projection process 

2s to generate the display image for each of the display points 
of view 100 corresponding to display elements 82 located 
away from the location of the principle display point of view 
98 where those hidden objects and sides of objects are 
visible. 

30 Hie reverse ray projection process is illustrated in FIG. 13 
wherein five display elements 82 in the display panel 80 are 
shown. To the right of the display panel 80 is shown the 
location of the viewer 86 positioned in front of and looking 
at the panel. Display rays 94 are also shown being output 

35 from the display elements 82 in the direction of the viewer 
86. As mentioned previously, each display element will 
project using display rays 94. an image of the scene 14 
acquired by the image acquisition device 12 (see FIGS. 1 
and 4). Accordingly, because the ideal number of laser 

40 sources 92 preferably equals the number of orientations of 
the scan ray 24, each display ray 94 output from the display 
elements 82 corresponds inversely with an orientation of the 
scan ray 24 imaging the scene 14. The scene can therefore 
be imagined mathematically to exist in three dimensions (as 

45 illustrated by broken lines) immediately behind the display 
panel 80 which then takes on the appearance of a window 
through which the scene 14 may be 

A better understanding of this effect may be obtained by 
processing the non-standard image obtained by the image 

50 acquisition device 12 of the three objects 16(1>-16(3) in 
FIG. 5. The non-standard image (FIG. 18) received by the 
display processor 84 includes a pixel field 204 having image 
information (image and distance value pairs 212) for each of 
three objects 16(1)-16(3) located along a particular orien- 

55 tation of principle scan ray 2Ap. The display element 82 at 
the principle point of view 98 displays the image informa- 
tion concerning only the nearest object 16(1) visible from 
that point of view since the other objects 16(2)-16(3) are 
hidden from the corresponding principle points of view 24 

60 and 98. In accordance with the reverse ray projection 
process, the measured distances to the objects 16(1)-16(3) 
from the principle point of view 24 along scan ray 24p are 
processed to detennine, for each of the display points of 
view 100 away from the principle display point of view 98. 

65 and further for each picture element within each of the points 
of view 98. orientations of the non-principle scan rays 24n 
to the objects. This is necessary because from the display 
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points of view 100 away from the principle display point of 
view 98, all of the objects 16(1)-16(3) are visible. Images at 
each of the display points of view 100 are thus generated 
picture element by picture element from the received non- 
standard image to include display rays 94 (forming display 5 
images) corresponding inversely to the determined scan rays 
24n. This reverse ray projection process is repeated for all of 
the pixel fields 204 in the image (having the format of FIGS. 
16 and 18) to generate an image comprising inversely 
corresponding display rays 94 for each point of view 100 on 1Q 
the display panel 80. and thus display image information 
concerning the nearest object visible from the correspond- 
ing point of view 100. 

Because adjacent or nearly adjacent display elements 82 
display slightly different perspectives of the scene, parallax 15 
is induced with respect to the viewer 86 of the display panel 
80 giving rise to a three-dimensional effect As the viewer 86 
moves laterally across the display panel 80, different points 
of view concerning the scene, and thus hidden objects and 
the sides of objects, are brought into view via the lenses 88 ^ 
(FIGS. 10-11) to produce a true three-dimensional effect 
This is illustrated in FIG. 13 wherein the left and right eyes 
of the viewer 86 see different images being displayed by the 
display elements 82 at the points of view 98 and 100 
corresponding to the perspective of each eye because dif- M 
ferent display rays 94 are being viewed through the lenses 
88. 

Reference is now made to FIG. 14 wherein there is shown 
an orthogonal view of the installation of the imaging and 
display system of the present invention in a room 110 for use 30 
in a teleconferencing application. The room 110 includes a 
combination display and imaging wall 112 comprising a 
plurality of spaced apart stereo cameras 38 mounted within 
a display panel 80 (including arrayed display elements 82). 
For teleconferencing purposes, the room 110 includes a 35 
conference Cable 114 having an end 116 abutting against the 
wall 112. In operation, the stereo cameras 38 function as 
described above with respect to FIGS. 1-6 to acquire 
three-dimensional image information concerning the room 
110 and the persons located therein (and perhaps seated 40 
around the table 114). This image information is transmitted 
to another location for three dimensional viewing using a 
display panel 80 comprising all or a portion of a wall 112. 
Similarly, the other location sends three-dimensional imag- 
ing information to the room 110 for viewing by persons in 45 
the room using the display panel 80 of the wall 112. If the 
other location is configured with a similar room size and 
table shape and position, the display panel 80 will give to 
viewers in either room the impression of the three- 
dimensional extension of the table 114 into the wall 112 30 
adjoining to the table in the other location. Three- 
dimensional viewing of not only the table, but also the 
persons in the other location is accordingly experienced with 
the display panel 80. A similar effect is presented to those at 
the other location with respect to the room 110. 55 

Referring now to FIG. 15A. several rooms 110 equipped 
with the imaging and display system of the present invention 
are shown. These rooms are of differing configuration, and 
are located in a plurality of different locations. End rooms 
UOe include the conference table 114 abutting into the 60 
imaging and display wall 112. The middle room 110m 
includes a display wall 112 at each end of the conference 
table 114. Each display wall 112 included in a room 110 
comprises a whole wall display panel 80 with included 
spaced apart stereo cameras 38. 65 

The cameras 38 image their rooms 110 for display on the 
display panels 80 of the other rooms. For example, the 
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images taken by the stereo cameras 38 of the left end room 
100* are displayed by the left display panel 801 of the 
middle room 100m. This true three-dimensional image of the 
left end room 100? is included in the image taken by the 
stereo cameras 38 on the right wall 112r of the middle room 
100m and displayed by the display panel SO of the right end 
room 110*. By using the various included stereo cameras 38, 
and displaying the images on the appropriate walls 112 of 
the rooms 110, a 'Virtual** conference room 116, as shown in 
FIG. 15B, is generated wherein an appearance of one large 
conference room is presented to the persons located in the 
various included rooms 110 due to the true three- 
dimensional imaging and display capabilities of the system 
of the present invention, It will, of course, be understood that 
the individual rooms and the virtual conference room can 
take on any one of a number of shapes. 

Although a preferred embodiment of the method and 
apparatus of the present invention has been illustrated in the 
accompanying Drawings and described in the foregoing 
Detailed Description, it will be understood that the invention 
is not limited to the embodiment disclosed, but is capable of 
numerous rearrangements, modifications and substitutions 
without departing from the spirit of the invention as set forth 
and defined by the following claims. 

What is claimed is: 

1. A true three-dimensional imaging and display system, 
comprising: 

an image acquisition device including: 
means for taking images of a scene from a plurality of 
spaced apart taken points of view, each taken image 
including image information for objects in the scene 
that are visible from the taken point of view of that 
image; and 

means for converging the taken images to generate a 
non-standard image of the scene from a principle 
point of view, the generated non-standard image 
including image information for the objects imaged 
in the scene from any of the taken points of view but 
not necessarily visible from the principle point of 
view; and 

an image display device connected to receive the non- 
standard image from the image acquisition device and 
including: 

means for processing the received non-standard image 
to generate a plurality of output images of the scene, 
each generated output image being from a slightly 
different display point of view, including points of 
view different from the taken and principle points of 
view, and showing the imaged objects in the scene 
visible from only mat display point of view; and 

a display panel including an arrayed plurality of image 
displays, wherein each image display is associated 
with a different one of the display points of view and 
is operable to display the generated output image 
corresponding to that display point of view, and 
wherein each image display further includes a lens 
for fanning out the output image displayed by that 
image display. 

2. The system of claim 1 wherein the means for taking 
comprises a plurality of stereo cameras located at the 
plurality of spaced apart taken points of view. 

3. The system of claim 2 wherein each stereo camera 
comprises: 

a left imaging camera for outputting a left image of the 
scene; 

a right imaging camera for outputting a right image of the 
scene; and 
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means for processing the left and right images output by 
the imaging cameras to generate the taken image of the 
scene from the taken point of view. 

4. The system of claim 3 wherein the means for process- 
ing comprises: 3 

means for comparing the left and right images; and 
means for determining from the comparison of the images 

a distance from the stereo camera to each object imaged 

in the left and right images of the scene. 

5. The system of claim 1 wherein each taken image of the 10 
scene comprises a plurality of picture elements, each picture 
element defined by an image value for the object imaged in 
that picture element and a distance from the taken point of 
view to that imaged object 

6. The system of claim 5 wherein the means for converg- 
ing comprises: 

means for converting each picture element in the taken 
images to a corresponding picture element in the non- 
standard image; 

means for converting the distances in the taken image to ^ 
distances to the imaged object from the principle point 
of view; and 

means for associating the converted distances to the 
corresponding picture elements. 

7. The system as in claim 6 wherein the means for ^ 
converting each picture element comprises means for iden- 
tifying the picture element in the non-standard image in 
which the object imaged from the taken point of view would 

be imaged from the principle point of view. 

8. The system of claim 1 wherein the non-standard image 
of the scene comprises a plurality of picture elements with 
each picture element being associated with a particular 
orientation of a scan ray emanating from the principle point 
of view, and wherein each picture element is defined by an 
image value and a distance from the principle point of view 35 
for each imaged object in the scene located along the scan 
ray associated with that picture element 

9. The system of claim 8 wherein the means for process- 
ing the received non-standard image comprises: 

means for converting the distances in the non-standard 40 
image to distances to the imaged objects from each of 
the display points of view; 

means for converting each picture element in the non- 
standard image to a corresponding picture element in 
each of the output images; and 45 

means for associating the converted distances to the 
corresponding picture elements, the closest converted 
distances for each picture element identifying the 
objects in the scene visible from the display point of 
view. 50 

10. The system of claim 1 wherein each output image 
comprises a plurality of picture elements, each picture 
element defined by an image value for an object displayed 
by that picture element. 

11. The system as in claim 10 wherein the image display 55 
comprises a plurality of laser sources corresponding to the 
plurality of picture elements in the output image* the laser 
sources emitting a display ray having the image value, the 
emitted display ray refracted by the lens. 

12. The system as in claim 11 wherein each laser source 60 
comprises a sub-array of mini-laser sources. 

13. The system as in claim 11 wherein a surface of the lens 
further includes a plurality of mini-lenses thereon for 
spreading the fanned display rays emitted from the laser 
sources. 65 

14. The system as in claim 1 wherein the fanning out of 
the output image by the lens restricts the amount of the 
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output image that is visible at each display element to a 
viewer of the display panel from a particular point of view 
with respect to the display panel, the slightly different output 
images displayed at the plurality of display elements induc- 
ing viewer parallax and three-dimensional perception of the 
displayed image of the scene. 

15. The system as in claim 1 further including an image 
storage device for storing images output by the image 
acquisition device. 

16. The system as in claim 15 wherein the image storage 
device further includes an image processing unit for manipu- 
lating images output by the image acquisition device. 

17. The system as in claim 1 wherein the means for taking 
images comprises means for scanning a three-dimensional 
space comprising the scene to acquire image information for 
the included objects. 

18. A three-dimensional image processing system, com- 
prising: 

means for generating a true three-dimensional non- 
standard image of a scene comprising a plurality of 
picture elements with each picture element defined by 
an image value and a distance from a principle point of 
view to each object in the scene located along a scan 
ray associated with that picture element; 

means for processing the image values and distances in 
the generated non-standard image of the scene to 
generate a plurality of output images of the scene from 
slightly different display points of view, each output 
image displaying the scene and the objects visible from 
the display point of view of the output image; and 

means for displaying the plurality of output images for 
true three-dimensional perception of the scene. 

19. The system of claim 18 wherein the means for 
generating comprises means for scanning in raster fashion a 
three-dimensional space comprising the scene. 

26. The system as in claim 18 wherein the means for 
generating the non-standard image comprises an image 
acquisition device, comprising: 
means for taking images of a scene from a plurality of 

spaced apart taken points of view; and 
means for composing the taken images to the principle 
point of view to produce the non-standard image of the 
scene from the principle point of view. 
2L The system as in claim 20 wherein the means for 
taking comprises a plurality of spaced apart stereo cameras. 

22. The system as in claim 20 wherein the means for 
composing comprises: 

means for converting image values and distances to 
objects from the taken points of view to image values 
and distances to objects from the principle point of 
view; and 

means for associating the converted image values and 
distances for objects in the scene located along the scan 
rays with the picture element in the non-standard image, 
associated with that scan ray. 

23. The system as in claim 18 wherein the means for 
generating a non-standard image comprises an image gen- 
eration device for creating and outputting the non-standard 
image of the scene from the principle point of view. 

24. The system as in claim 23 wherein the means for 
generating further comprises means for manipulating previ- 
ously acquired images to generate and output the non- 
standard image. 

25. The system as in claim 18 wherein each output image 
generated by the means for processing comprises a plurality 
of picture elements with each picture element defined by an 
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image value and a distance from the display point of view to 
the nearest object in the scene. 

26. The system as in claim 19 wherein the means for 
processing comprises: 

means for converting the image values and distances from 5 
the principle point of view to image values and dis- 
tances from each of the display points of view; and 

means for associating the converted image values and 
distances for objects in the scene with picture elements 
in the output images. 10 

27. The system as in claim 18 wherein the means for 
displaying comprises: 

a display panel; and 

an arrayed plurality of display elements on the display i5 
panel each display element displaying one of the 
plurality of output images. 

28. The system as in claim 27 wherein each display 
element comprises: 

a display device for displaying an output image; and ^ 
a lens for fanning out the displayed output image, the lens 
accordingly restricting the amount of the output image 
displayed by the display device that is visible to a 
viewer from a given viewing point of view with respect 
to the display panel. 2 s 

29. The system as in claim 28 wherein the display device 
comprises an arrayed plurality of laser sources, one laser 
source for each picture element in the output image received 
by the display device, each laser source emitting a display 
ray having the image value of that picture element ^ 

30. The system as in claim 29 wherein each laser source 
comprises an arrayed plurality of mini-laser sources. 

31. The system as in claim 28 wherein each lenses 
includes a plurality of mini-lenses positioned on a surface of 
the lens. 35 

32. A method for true three-dimensional imaging of a 
scene including a plurality of objects, comprising the steps 
of: 

taking a plurality of images of the scene and its included 
objects from a plurality of spaced apart taken points of 40 
view; 

converting the plurality of taken images to a principle 
point of view; and 

combining the converted images to generate a non- 
standard image of the scene and its included objects 45 
from the principle point of view, the non-standard 
image including image information for the imaged 
objects in the scene including those objects not neces- 
sarily visible from the principle point of view. 

33. The method of claim 32 wherein the step of taking 50 
comprises the step of stereo imaging the scene from each of 
the taken points of view to determine image information for 
the objects in the scene including distances from the taken 
points of view to each imaged object 

34. The method as in claim 33 wherein the step of 55 
converting comprises the step of converting the distances 
from the taken points of view to each imaged object to 
distances from the principle point of view to each imaged 
object 

35. The method as in claim 34 wherein the step of 60 
combining comprises the step of associating, for each 
imaged object the imaged object and the converted distance 

to that object from the principle point of view with a picture 
element in the non-standard image. 

36. The method as in claim 35 wherein each picture 65 
element in the non-standard image is associated with a 
particular orientation of a scan ray emanating from the 
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principle point of view, and the step of associating comprises 
the step of associating the image information for each 
imaged object located along the scan ray with the picture 
element associated with that scan ray. 

37. A method for true three-dimensional display of an 
image of a scene including a plurality of objects, the image 
being from a principle point of view and including image 
information for the objects both visible and hidden from the 
principle point of view, comprising the steps of: 

converting the image from the principle point of view to 
generate a plurality of output images of the scene, each 
output image being from a slightly different display 
point of view; 

simultaneously displaying the plurality of output images 
of the scene; and 

restricting viewer perception of portions of each of the 
displayed plurality of output images to induce parallax 
in the displayed scene and true three-dimensional per- 
ception of the objects therein. 

38. The method as in claim 37 wherein the step of 
converting comprises the step of converting the image 
information from the principle point of view to image 
information from each of the display points of view. 

39. The method as in claim 37 wherein the step of 
simultaneously displaying comprises the step of displaying 
the output images in an arrayed fashion. 

40. The method as in claim 37 wherein the step of 
restricting comprises the step of fanning out through a lens 
each of the displayed output images. 

41. A true three-dimension image acquisition device, 
comprising: 

a set of cameras for taking images of a scene from a 
plurality of spaced apart taken points of view, each 
taken image including image information for objects in 
the scene that are visible from the taken point of view 
of that image; and 

an image processor for converging the taken images to 
generate a non-standard image of the scene from a 
principle point of view, the generated non-standard 
image including image information for the objects 
imaged in the scene from any of the taken points of 
view but not necessarily visible from the principle point 
of view. 

42. The device of claim 41 wherein the set of cameras 
comprise a plurality of stereoscopic cameras for taking 
stereo images of the scene from the taken points of view. 

43. The device of claim 42 wherein each stereoscopic 
camera comprises: 

a left camera; 

a right camera; and 

means for processing images obtained from the left and 
right cameras to determine distances to each object 
viewed by the left and right cameras and output the 
stereo image of the scene comprising a plurality of 
picture elements, each picture element defined by an 
image value and the determined distance. 

44. The device of claim 41 wherein the generated non- 
standard image output by the image processor comprises a 
plurality of picture elements, each picture element defined 
by an image value and a distance value to each object in the 
scene located along a scan ray emanating from the principle 
point of view whose orientation corresponds to that picture 
element. 

45. The device of claim 41 wherein the taken image 
comprises a plurality of picture elements, each picture 
element defined by an image value and distance value to a 
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closest object in the scene visible from die taken point of 
view, and wherein the image processor includes; 
means for converting the picture element of the taken 
image to a corresponding picture element of the non- 
standard image; 5 
means for converting the distance value from the taken 
point of view to a new distance value from the principle 
point of view; and 
means for associating the image value of new distance ]0 
value for picture element of the taken image to its 
corresponding picture element of the non-standard 
image. 

46. A true three-dimension image display device, com- 
prising: 15 

an image processor for converting a non-standard image 
of a scene including a plurality of picture elements 
wherein each picture element includes image informa- 
tion for objects within the scene mat are both visible 
and not visible from a principle point of view to 20 
generate a plurality of output images of the scene, each 
generated output image being from a slightly different 
display point of view and including a plurality of 
picture elements wherein each picture element includes 
image information for objects in the scene visible from 
only that display point of view; and 
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a display panel including an arrayed plurality of image 
displays, wherein each image display is associated with 
a different display point of view and is operable to 
display the generated output image corresponding to 
that display point of view, and wherein each image 
display further includes a lens for fanning out the 
output image displayed by mat image display. 

47. The device of claim 46 wherein each image display 
comprises a plurality of laser sources corresponding to the 
picture elements in the output image, the laser sources 
outputting a display ray in accordance with the image 
information of the corresponding picture element of the 
output image. 

48. The device of claim 47 wherein each laser source 
comprises an arrayed plurality of mini-laser sources. 

49. The device of claim 46 wherein the fanning out of the 
output image by the lens restricts the amount of the output 
image that is visible at each display element to a viewer of 
the display panel from a particular point of view with respect 
to the display panel, the slightly different output images 
displayed at the plurality of display elements inducing 
viewer parallax and three-dimensional perception of the 
displayed image of the scene. 

50. The device of claim 46 wherein the lens further 
comprises a plurality of mini-lenses on a surface of the lens. 

***** 
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